Mention Model for Learning Rules from Incomplete Examples

Introduction. We are motivated by the problem of learning rules from naturally available data sources such as natural language texts, web pages, and medical databases. At first, learning rules from natural sources like the web seems to consist of extracting specific facts followed by data mining of rules. Unfortunately, however, there are two major obstacles to fully realizing the dream of unlimited learning of general rules from natural sources. First, natural data sources such as texts and medical histories are radically incomplete in that only a tiny fraction of all true facts are ever mentioned. Perhaps more discouragingly, natural sources are systematically biased in what is mentioned. For example, news stories are biased towards newsworthiness, which correlates with rarity or novelty, sometimes referred as “the man bites dog phenomenon.” In previous work, we introduced the notion of a mention model which models the observation process of an agent generating the data. We showed the effectiveness of an implicit mention model in learning rules by adapting the scoring function which is used to score the hypothesized rules [Doppa et al., 2010]. While implicit mention models are very simple, their usefulness is debatable especially when the mention model is quite complicated. In this work, we propose a generative approach to explicitly model the mention process of data. We propose an iterative EM style algorithm to learn the parameters of our model. We demonstrate the usefulness of the proposed explicit mention model on both synthetic and real-world datasets. Explicit Mention Model. Extracted facts from natural language texts can be typically represented using a set of interrelated predicates. Therefore, it is possible to propose a set of inaccurate rules to predict each predicate from the remaining ones. Our goal then is to construct a probabilistic mention model that captures what facts are mentioned and extracted by the extractor from the text given the true facts about the world. Given some extracted facts, the learning agent inverts the mention model to infer a distribution over sets of true facts. An inductive program can then be used to infer general rules from distributions over true facts. Figure 1 shows the idea of explicit mention model and how the mention observations interacts with the true facts Learning Explicit Mention Model. An iterative approach to learn explicit mention model in an Expectation Maximization (EM) setting is shown in Algorithm 1. We use a relational data mining algorithm called FARMER [Nijssen and Kok, Figure 1: Explicit Mention Model Interaction between facts and the mention observations